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THREE-DIMENSIONAL MODELING APPARATUS , METHOD, AND MEDIUM, AND 
THREE-DIMENSIONAL SHAPE DATA RECORDING APPARATUS, METHOD, AND 

MEDIUM 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a three-dimensional 
modeling technique for producing three-dimensional shape data of 
a target from a plurality of image data of the target, and in 
particular to three-dimensional modeling apparatus, method, and 
medium, and three-dimensional shape data recording apparatus, 
method, and medium. 

2 . Description of the Related Art 

Conventionally, it is widely known to use a silhouette method 
where three-dimensional shape is estimated from silhouette 
information of a target, a triangulation method using a laser light, 
or a space coding method where a plurality of black-and-white 
patterns are projected, as a method for producing various 
three-dimensional images . 

In the silhouette method, the shape data is produced as 
follows. For example, the target is placed on a rotational table, 
a plurality of images are captured with a suitable angle difference , 
and a silhouette image of the target is determined for each of the 
images . Here, by using a background plate such as a blue background 
during the capturing process, silhouette images can be easily and 
stably obtained using a widely-known chroma key technique. 

If the positional relationship between the rotational table 



and the capturing camera is accurately known in advance, a 
silhouette can be projected to the assumed three-dimensional space 
on the rotational table from each viewpoint. The space where all 
silhouettes are projected is extracted as the object region. In 
this manner, the three-dimensional shape of the target is 
estimated. 

The triangulation method using a laser light, on the other 
hand, uses a laser light irradiating device having a basic 
structure as shown in Fig. 15. Specifically, as shown in Fig. 15, 
a beam shape laser light is converted to a slit light by a 
cylindrical lens and the direction of the slit light is altered 
using a galvano-mirror or a polygon mirror, to allow a two- 
dimensional scan. 

The laser light irradiated onto the target in this manner is 
then captured by an image inputting section such as a camera. The 
laser light has a sufficiently larger intensity than a typical 
illumination light, and thus, the point (or line) of maximum 
luminance can be easily extracted from the input image. The 
positional relationship between the laser light source and the 
image inputting section is accurately measured in advance. If the 
angle of the mirror when the image is input is accurately known, 
the distance from the camera or the laser light source to the target 
can be calculated using the principle of triangulation. 

In a space coding method, a plurality of black-and-white 
patterns such as the patterns shown in Fig. 9 are projected onto 
a target and the target is captured. Then, for each image, it is 
determined which of the black or white pattern is projected on each 




pixel. It is known that the accuracy can be improved by comparing 
the projected image with an image projected by a pattern obtained 
by inverting the patterns described above. 

In this manner, when seven patterns are projected, for example, 
5 the image can be divided into a total of 128 strip regions. If 
the direction of the pattern and the direction of the camera towards 
the irradiating section are approximately perpendicular, the 
pattern sequence of each pixel and the direction of the light from 
the pattern projecting section can be associated. Thus, the 
10 distance between the camera or the laser light source to the target 
p can be calculated based on the principle of triangulation . 

,J In the above silhouette method, the three-dimensional shape 

can be estimated once the background and the target are separated 
!L and silhouette information is obtained. Thus, there is no 

yj 15 limitation on the material or the color of the target. However, 
there is a problem that the three-dimensional shape of any shape 
N 5 that does not show a silhouette outline cannot be accurately 

estimated. Specific examples of such a shape include the internal 
section of a cup. This portion always corresponds to the internal 
20 silhouette of another portion and cannot be observed as a 
silhouette outline, and the three-dimensional shape cannot be 
accurately estimated by the silhouette method. 

In the triangulation method using a laser light and space 
coding method, on the other hand, it is fundamentally necessary 
25 to irradiate light onto the target and distinguish between the 
irradiated and non-irradiated sections. However, in a black 
portion, for example, the hair portion of a person, this 
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distinction is very difficult. Therefore, in the triangulation 
method using laser light and space coding method, it is difficult 
to obtain the three-dimensional shape information for materials 
and colors that absorb light. There is a solution to this problem 
5 using a very strong laser light, but when the target is a person, 
there will be a problem regarding safety. 

In addition, because a plurality of patterns must be projected 
in time division in the space coding method, a high-speed data 
inputting operation is difficult. Similarly, in the 



yg 10 triangulation method using a laser light, because only one laser 

,p beam can be irradiated and this one laser beam must be scanned, 

p there is a problem that a high-speed data inputting operation is 

?A difficult. 



dimensional modeling apparatus for generating three dimensional 
shape data of a three dimensional object, comprising projection 
means for projecting pattern light onto the three dimensional 

20 object; image input means for inputting an image of the three 
dimensional object; rough shape calculation means for calculating 
a rough shape from the image; detailed shape calculation means for 
calculating a detailed shape from the image; and three dimensional 
shape data generating means for generating three dimensional shape 

25 data of the three dimensional object based on the rough shape and 
the detailed shape. 
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SUMMARY OF THE INVENTION 



According to the present invention, there is provided a three 
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Further, there is provided a three dimensional modeling 
apparatus for generating three dimensional shape data of a three 
dimensional object , comprising projection means for projecting 
pattern light onto the three dimensional object; image input means 
5 for inputting an image of the three dimensional object; rough shape 
input means for inputting a rough shape of the three dimensional 
object; detailed shape calculation means for calculating a 
detailed shape from the image; and three dimensional shape data 
generating means for generating three dimensional shape data of 
10 the three dimensional object based on the rough shape and the 

til 

WW 

=JE detailed shape, 

Q Still further, there is provided a three dimensional modeling 

h4 apparatus for generating three dimensional shape data of a three 

rj dimensional object, comprising image input means for inputting an 

g 15 image of the three dimensional object, the image being obtained 
3 by projecting pattern light onto the three dimensional object; 

rough shape calculation means for calculating a rough shape from 
the image; detailed shape calculation means for calculating a 
detailed shape from the image; and three dimensional shape data 
20 generating means for generating three dimensional shape data of 
the three dimensional object based on the rough shape and the 
detailed shape. 

Also, there is provided a three dimensional modeling 
apparatus for generating three dimensional shape data of a three 
25 dimensional object, comprising image input means for inputting an 
image of the three dimensional object, the image being obtained 
by projecting pattern light onto the three dimensional object; 
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rough shape input means for inputting a rough shape of the three 
dimensional object; detailed shape calculation means for 
calculating a detailed shape from the image; and three dimensional 
shape data generating means for generating three dimensional shape 
5 data of the three dimensional object based on the rough shape and 
the detailed shape. 

Preferably, in the above three dimensional modeling apparatus, the 
three dimensional shape data generating means may determine the 
final three dimensional shape data based on the following rule; 
g 10 if the detailed shape exits inside the rough shape, then the 
4*. detailed shape is the final three dimensional shape data of said 

P object, otherwise the rough shape is taken as the final three 

^ dimensional shape data of said object. 

3 Also, in the above three dimensional modeling apparatus, the 

3 15 three dimensional shape data generating means may obtain a region 
I where the object surface can exist based on the rough shape, to 

generate three dimensional shape data of the three dimensional 
object based on the rough shape, the detailed shape, and the region 
where the object surface can exist. 
20 Further, in the above three dimensional modeling apparatus, 

the detailed shape calculation means may obtain a plurality of 
object surface location candidates from the input image. 

Still further, preferably, in the above three dimensional 
modeling apparatus, the detailed shape calculation means may set 
25 a region where the object surface cannot exist in a portion inside 
the rough shape and may determine a region inside the rough shape 
other than the region where the object surface cannot exist as a 




6 




region where the object surface can exist, to generate three 
dimensional shape data of the three dimensional object based on 
the rough shape, the detailed shape, and the region where the object 
surface can exist. 
5 Preferably, in the above three dimensional modeling apparatus, 

the three dimensional shape data generating means may determine 
a final object surface location candidate from a plurality of 
object surface location candidates based on a region where the 
object surface can exist and may generate three dimensional shape 

W 10 data of the three dimensional object based on the final object 

HN surface location candidate. 

0 Also, in the above three dimensional modeling apparatus , the 
rough shape may be calculated by the rough shape calculation means 

3 based on silhouette information obtained from the object image 

3 15 input by the image input means. 

1 According to the present invention, there is provided a three 
dimensional shape data recording apparatus for recording three 
dimensional shape data of a three dimensional object, comprising 
three dimensional shape data generation means for generating three 

20 dimensional shape data of the three dimensional object; three 
dimensional shape data reliability assigning means for assigning 
a data reliability evaluation value to each of the three 
dimensional shape data; and three dimensional shape data recording 
means for recording three dimensional shape data to which the 

25 reliability evaluation value is assigned in a medium. 

Also, there is provided a three dimensional shape data 
recording apparatus for recording three dimensional shape data of 
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a three dimensional object, comprising three dimensional shape 
data capturing means for capturing three dimensional shape data 
of the three dimensional object; three dimensional shape data 
reliability assigning means for assigning a data reliability 
evaluation value to each of the three dimensional shape data; and 
three dimensional shape data recording means for recording three 
dimensional shape data to which the reliability evaluation value 
is assigned in a medium. 

Still further, there is provided a three dimensional modeling 
method for generating three dimensional shape data of a three 
dimensional object, comprising the steps of projecting pattern 
light onto the three dimensional object; inputting an image of the 
three dimensional object; calculating a rough shape from the image; 
calculating a detailed shape from the image; and generating three 
dimensional shape data of the three dimensional object based on 
the rough shape and the detailed shape. 

Moreover, there is provided a three dimensional modeling 
method for generating three dimensional shape data of a three 
dimensional object, comprising the steps of: projecting pattern 
light onto the three dimensional object; inputting an image of the 
three dimensional object; capturing a rough shape of the three 
dimensional object; calculating a detailed shape from the image; 
and generating three dimensional shape data of the three 
dimensional object based on the rough shape and the detailed shape. 

Still further, there is provided a three dimensional modeling 
method for generating three dimensional shape data of a three 
dimensional object, comprising the steps of inputting an image of 
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the three dimensional object , the image being obtained by 

projecting pattern light onto the three dimensional object; 

calculating a rough shape from the image; calculating a detailed 

shape from the image; and generating three dimensional shape data 
5 of the three dimensional object based on the rough shape and the 

detailed shape. 

Further, there is provided a three dimensional modeling 

method for generating three dimensional shape data of a three 
m dimensional object , comprising the steps of inputting an image of 

10 the three dimensional object, the image being obtained by 
ifj projecting pattern light onto the three dimensional object; 

*f capturing a rough shape of the three dimensional object; 

calculating a detailed shape from the image; and generating three 
% dimensional shape data of the three dimensional object based on 

f 15 the rough shape and the detailed shape. 

as 

In the above three dimensional modeling method, the step of 
generating three dimensional shape data may include determining 
the final three dimensional shape data based on the following rule; 
if the detailed shape exits inside the rough shape, then the 

20 detailed shape is the final three dimensional shape data of said 
object, otherwise the rough shape is taken as the final three 
dimensional shape data of said object. 

Further, in the above three dimensional modeling method, the 
step of generating three dimensional shape data may include 

25 obtaining a region where the object surface can exist based on the 
rough shape, to generate three dimensional shape data of the three 
dimensional object based on the rough shape, the detailed shape, 
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and the region where the object surface can exist. 

Preferably, in the above three dimensional modeling method, 
the step of calculating detailed shape may include obtaining a 
plurality of object surface location candidates from the input 
5 image . 

Also, in the above three dimensional modeling method, the step 
of calculating detailed shape may include setting a region where 
the object surface cannot exist in a portion inside the rough shape 
and determining a region inside the rough shape other than the 
10 region where the object surface cannot exist as a region where the 
object surface can exist, to generate three dimensional shape data 
J=f of the three dimensional object based on the rough shape, the 

^ detailed shape, and the region where the object surface can exist. 

;!j Further, in the above three dimensional modeling method, the 

Q 15 step of generating three dimensional shape data may include 
P determining a final object surface location candidate from a 

plurality of object surface location candidates based on a region 
where the object surface can exist and generating three dimensional 
shape data of the three dimensional object based on the final object 
20 surface location candidate. 

Preferably, in the above three dimensional modeling method, 
the rough shape may be calculated by the rough shape calculation 
means based on silhouette information obtained from the object 
image input by the step of inputting an image. 
25 According to the present invention, there is also provided 

a three dimensional shape data recording method for recording three 
dimensional shape data of a three dimensional object, comprising 
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the steps of generating three dimensional shape data of the three 
dimensional object; assigning a data reliability evaluation value 
to each of the three dimensional shape data; and recording three 
dimensional shape data to which the reliability evaluation value 
5 is assigned in a medium. 

Further , there is provide a three dimensional shape data 
recording method for recording three dimensional shape data of a 
three dimensional object , comprising the steps of capturing three 
^ dimensional shape data of the three dimensional object; assigning 

|r 10 a data reliability evaluation value to each of the three 

ess 

J? dimensional shape data; and recording three dimensional shape data 

H to which the reliability evaluation value is assigned in a medium. 

^ Still further, there is provided a medium having a program 

y recorded therein for causing a computer to generate three 

O 15 dimensional shape data of a three dimensional object, the program 
Q comprising the steps of controlling pattern light projection 

section connected to the computer for projecting pattern light onto 
the three dimensional object; inputting an image of the three 
dimensional object; calculating a rough shape from the image; 
20 calculating a detailed shape from the image; and generating three 
dimensional shape data of the three dimensional object based on 
the rough shape and the detailed shape. 

Moreover, there is provided a medium having a program recorded 
therein for causing a computer to generate three dimensional shape 
25 data of a three dimensional object, the program comprising the 
steps of controlling a pattern light projection section connected 
to the computer for projecting pattern light onto the three 
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dimensional object; inputting an image of the three dimensional 
object; capturing a rough shape of the three dimensional object; 
calculating a detailed shape from the image; and generating three 
dimensional shape data of the three dimensional object based on 
5 the rough shape and the detailed shape. 

Still further, there is provided a medium having a program 
recorded therein for causing a computer to generate three 
dimensional shape data of a three dimensional object, the program 
comprising the steps of : inputting an image of the three 
^ 10 dimensional object, which is obtained by projecting pattern light 
,7j onto the three dimensional object; calculating a rough shape from 

the image; calculating a detailed shape from the image; and 
f~ generating three dimensional shape data of the three dimensional 

rz.; object based on the rough shape and the detailed shape, 

.jf 15 There is also provided a medium having a program recorded 

jhf therein for causing a computer to generate three dimensional shape 

data of a three dimensional object, the program comprising the 
steps of inputting an image of the three dimensional object, which 
is obtained by projecting pattern light onto the three dimensional 
20 object; capturing a rough shape of the three dimensional object; 
calculating a detailed shape from the image; and generating three 
dimensional shape data of the three dimensional object based on 
the rough shape and the detailed shape. 

In the above medium, the step of generating three dimensional shape 
25 data may include determining the final three dimensional shape data 
based on the following rule; if the detailed shape exits inside 
the rough shape, then the detailed shape is the final three 
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dimensional shape data of said object, otherwise the rough shape 
is taken as the final three dimensional shape data of said object. 

Also, in the above medium, the step of generating three 
dimensional shape data may include obtaining a region where the 
5 object surface can exist based on the rough shape, to generate three 
dimensional shape data of the three dimensional object based on 
the rough shape, the detailed shape, and the region where the object 
surface can exist. 
^ Preferably, in the above medium, the step of calculating a 

It 10 detailed shape may include obtaining a plurality of object surface 
J! location candidates from the input image. 

^ Further, in the above medium, the step of calculating a 

detailed shape may include setting a region where the object 
J surface cannot exist in a portion inside the rough shape and 

f 15 determining a region inside the rough shape other than the region 
where the object surface cannot exist as a region where the object 
surface can exist, to generate three dimensional shape data of the 
three dimensional object based on the rough shape, the detailed 
shape, and the region where the object surface can exist. 
20 Further, preferably, in the above medium, the step of 

generating three dimensional shape data may include determining 
a final object surface location candidate from a plurality of 
object surface location candidates based on a region where the 
object surface can exist and generating three dimensional shape 
25 data of the three dimensional object based on the final object 
surface location candidate. 

Still further, in the above medium, the rough shape may be 
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calculated by the step of calculating a rough shape based on 
silhouette information obtained from the object image input by the 
step of inputting an image . 

According to the present invention, there is also provided 
5 a medium having a program recorded therein for causing a computer 
to record three dimensional shape data of a three dimensional 
object in the medium, the program comprising the steps of 
generating three dimensional shape data of the three dimensional 
^ object; assigning a data reliability evaluation value to each of 

10 the three dimensional shape data; and recording three dimensional 
j£T shape data to which the reliability evaluation value is assigned 

yT in the medium. 

There is also provided a medium having a program recorded 
therein for causing a computer to record three dimensional shape 
15 data of a three dimensional object in the medium, the program 
comprising the steps of capturing three dimensional shape data of 
the three dimensional object; assigning a data reliability 
evaluation value to each of the three dimensional shape data; and 
recording three dimensional shape data to which the reliability 
20 evaluation value is assigned in the medium. 

Also, there is provided a medium having three dimensional 
shape data of a three dimensional object recorded therein, the 
medium holding data reliability evaluation value information for 
each of the three dimensional shape data. 
25 Still further, the present invention provides a three 

dimensional modeling apparatus for generating three dimensional 
shape data of a three dimensional object, comprising: projection 
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means for projecting pattern light onto the three dimensional 
object; image input means for inputting an image of the three 
dimensional object; and three dimensional shape calculation means 
for calculating a three dimensional shape of the three dimensional 
5 object from the input image, wherein the three dimensional shape 
calculation means extracts a pattern projected region and a pattern 
border region in the input image to calculate the three dimensional 
shape of the three dimensional object based on these regions. 
, m In the three dimensional modeling apparatus according to the 

rf 10 present invention, the detailed shape calculation means may 
:J! extract a pattern projected region and a pattern border region in 

?*f the input image to calculate the detailed shape of the three 

dimensional object based on these regions. 
;% In the above three dimensional modeling apparatus, preferably, 

p 15 the pattern light may have a plurality of binary patterns. 
O In the above three dimensional modeling apparatus, preferably, 

the image input means may perform image input from at least two 
positions . 

Further, there is also provided a three dimensional modeling 
20 apparatus for generating three dimensional shape data of a three 
dimensional object, comprising projection means for projecting 
pattern light onto the three dimensional object; image input means 
for inputting an image of the three dimensional object; and three 
dimensional shape calculation means for calculating a three 
25 dimensional shape of the three dimensional object from the input 
image, wherein the image input means performs image input from at 
least two positions, the three dimensional shape calculation means 

15 



extracts a portion in the input image where color or density changes , 
and matching of portions where color or density changes is 
performed between input images, to calculate the three dimensional 
shape of the three dimensional object. 
5 In the three dimensional modeling apparatus according to the 

present invention, preferably, the image input means may perform 
image input where the images are captured at least two points at 
different locations, the detailed shape calculation means may 
extract a portion in the input image where color or density changes, 

10 and matching of portions where color or density changes may be 
performed between input images, to calculate the three dimensional 
shape of the three dimensional object. 

In the above three dimensional modeling apparatus, preferably, 
the image input means may perform image input from at least two 

15 different points at different locations, the detailed shape 
calculation means may extract a portion in the input image where 
color or density changes, and matching of potions where color or 
density changes may be performed between input images, to calculate 
the three dimensional shape of the three dimensional object. 

20 Preferably, in the above three dimensional modeling apparatus, 

the pattern light for projection may include multi-color patterns 
in which adjacent patterns have different colors having a hue 
difference of at least 90 degree or more, or a brightness difference 
of 0.3 or more, when the colors are represented in an HSV space. 

25 In the above three dimensional modeling apparatus, the 

matching of the portions where color or density changes may be 
obtained based on properties of the portions where color or density 
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changes, which are obtained from the input images. 

Further in the above three dimensional modeling apparatus, 
the property of the portion where color or density changes may be 
color information regarding portions located to the left and right 
of, or above and below, the portion where color or density changes 
in the input image. 

Still further, in the three dimensional modeling apparatus, 
averaging may be performed to the input images when the property 
is obtained. 

Also, in the three dimensional modeling apparatus, the 
averaging may be performed for each of divided regions of the input 
image . 

Preferably, the present invention provides a three 
dimensional modeling apparatus, further comprising moving means 
for moving the projection means and the image input means. 

Preferably, the moving means may move the projection means 
and the image input means based on the rough shape. 

According to the present invention, there is also provided 
a three dimensional modeling apparatus for generating three 
dimensional shape data of a three dimensional object, comprising: 
at least two three dimensional shape input means for obtaining a 
three dimensional shape of the three dimensional object; and moving 
means for moving the three dimensional shape input means, wherein 
the three dimensional shape input means have different three 
dimensional input accuracy. 

In the three dimensional modeling apparatus, preferably, the 
moving means may move the three dimensional shape input means 
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having higher three dimensional input accuracy based on three 
dimensional shape information obtained by the three dimensional 
shape input means having lower three dimensional input accuracy. 
According to the present invention, there is also provided 
5 a three dimensional modeling method for generating three 
dimensional shape data of a three dimensional object, comprising 
the steps of projecting pattern light onto the three dimensional 
object; inputting an image of the three dimensional object; and 
calculating a three dimensional shape of the three dimensional 

10 object from the input image, wherein the step of calculating the 
three dimensional shape includes extracting a pattern projected 
region and a pattern border region in the input image to calculate 
the three dimensional shape of the three dimensional object based 
on these regions. 

15 In the three dimensional modeling method according to the 

present invention, preferably, the step of calculating the 
detailed shape may include extracting a pattern projected region 
and a pattern border region in the input image to calculate the 
detailed shape of the three dimensional object based on these 

20 regions. 

In the three dimensional modeling method, preferably, the 
pattern light may have a plurality of binary patterns. 

Also, in the three dimensional modeling method, the step of 
inputting an image may include performing image input from at least 
25 two positions . 

Still further, according to the present invention, there is 
provided a three dimensional modeling method for generating three 
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dimensional shape data of a three dimensional object, comprising 
the steps of projecting pattern light onto the three dimensional 
object; inputting an image of the three dimensional object; and 
calculating a three dimensional shape of the three dimensional 
5 object from the input image , wherein the step of inputting an image 
includes performing image input where the images are captured at 
least two positions, the step of calculating the three dimensional 
shape includes extracting a portion in the input image where color 
13 or density changes, and matching portions where color or density 

?q 10 changes between input images, to calculate the three dimensional 
^ shape of the three dimensional object. 

J In the three dimensional modeling method according to the 

~~ present invention, preferably, the step of inputting an image may 

~ include performing image input from at least two points at 

f 15 different locations, the step of calculating a detailed shape 
includes extracting a portion in the input image where color or 
density change, and matching portions where color or density 
changes between input images, to calculate the three dimensional 
shape of the three dimensional object. 
20 In the three dimensional modeling method according to the 

present invention, preferably, the pattern light for projection 
may include multi-color patterns in which adjacent patterns have 
different colors having a hue difference of at least 90 degree or 
more, or a brightness difference of 0.3 or more, when the colors 
25 are represented in an HSV space. 

In the three dimensional modeling method, the matching of the 
portions where color or density changes may be obtained based on 
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properties of the portions where color or density changes, which 
are obtained from the input image. 

In the three dimensional modeling method, preferably, the 
property of the portion where color or density changes may be color 
5 information regarding portions located to the left and right of, 
or above and below, the portion where color or density changes in 
the input images. 

Further, in the three dimensional modeling method, averaging 
PS may be performed for the input images when the property is obtained. 

10 Still further, in the three dimensional modeling method, the 

T% averaging may be performed for each of divided regions of the input 

H images . 

^ According to the present invention, there is also provided 

Jjf a three dimensional modeling method for generating three 

j*f : 15 dimensional shape data of a three dimensional object, comprising: 
W at least two three-dimensional shape inputting steps of obtaining 

the three dimensional shape of the three dimensional object; and 
a step of moving a three dimensional shape input section, wherein 
the three dimensional shape inputting steps have different three 
20 dimensional input accuracy. 

Further, there is also provide a three dimensional modeling 
apparatus for generating three dimensional shape data of a three 
dimensional object, comprising first three- dimensional shape 
measuring means for obtaining rough shape information of the three 
25 dimensional object ; second three dimensional shape measuring means 
for obtaining detailed shape information of the three dimensional 
object; and a three dimensional shape data generating section for 
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generating the three dimensional shape data of the three 
dimensional object based on the rough shape information and the 
detailed shape information. 

According to the present invention, control means may be 
5 further provided for controlling the position of the second three 
dimensional shape measuring means, the control means being 
controlled based on the rough shape information obtained by the 
first three dimensional measuring means. 

Further, there is also provided a three dimensional modeling 

10 method for generating three dimensional shape data of a three 
dimensional object, comprising: a first three dimensional shape 
measuring step of obtaining rough shape information of the three 
dimensional object; a second three dimensional shape measuring 
step of obtaining detailed shape information of the three 

15 dimensional object; and a three dimensional shape data generating 
step of generating the three dimensional shape data of the three 
dimensional object based on the rough shape information and the 
detailed shape information. 

Further, the present invention provides a medium having a 

20 program recorded therein for causing a computer to generate three 
dimensional shape data of a three dimensional object, the program 
comprising a first three dimensional shape measuring step of 
obtaining rough shape information of the three dimensional object; 
a second three dimensional shape measuring step of obtaining 

25 detailed shape information of the three dimensional object; and 
a three dimensional shape data generating step of generating the 
three dimensional shape data of the three dimensional object based 
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on the rough shape information and the detailed shape information. 

BRIEF DESCRIPTION OF THE DRAWING 
Fig. 1 is a structural diagram showing one embodiment of the 
5 present invention. 

Fig. 2 is a flowchart showing one embodiment of the present 
invention . 

Fig. 3 is a diagram for explaining a perspective ratio. 
Fig. 4 is a diagram for explaining calibration (calculation 
10 of external parameters) . 

Fig. 5 is a diagram for explaining the principle of Hough 
transform. 

Fig. 6 is a diagram for explaining parameter determination 
in the Hough transform. 
15 Fig. 7 is a flowchart for rough shape calculation by a 

silhouette method. 

Fig. 8 is a diagram for explaining the shape reconstruction 
by the silhouette method. 

Fig. 9 shows an example of 7 types (A through G) of binary 
20 patterns. 

Fig. 10 is a diagram showing the principle of space coding 
method using the binary patterns. 

Fig. 11 is a diagram explaining the rough depth map, detailed 
depth map, and final depth map. 
25 Fig. 12 is a flowchart for another embodiment of the present 

invention . 

Fig. 13 is a diagram explaining the relationship between a 
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plurality of surface candidates and possible existence region for 
a surface based on the rough shape. 

Fig. 14 is an example of three-dimensional shape data with 
reliability information. 
5 Fig. 15 is a diagram explaining the basic structures of a laser 

light irradiating section. 

Fig. 16 is a diagram showing an example of inputting pattern 
projected images by two cameras. 
^ Fig. 17 is a diagram explaining an example of a code assignment 

10 error in the binary pattern . 
JT. Fig. 18 is a diagram showing another example of binary 

**f : patterns . 

r ~ Fig. 19 is a diagram explaining assignment of six types of 

!? coding regions . 

f 15 Fig . 20 is a diagram showing an example of a color code pattern . 

f Fig. 21 is a diagram showing , as a model, the depth value of 

the corresponding characteristic point candidate obtained using 

three cameras, A, B, and C. 

20 DESCRIPTION OF PREFERRED EMBODIMENT 

The preferred embodiments of the present invention will now 
be described. 

In the following description and in the appended claims, the 
term "image" includes a wide variety of images from digitized image 
25 information (image data) that can be processed by a computer to 
images that are printed on a 35 mm film or on print paper. 
(Description of the structure) 
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Fig . 1 conceptually shows a structure of the present invention . 
An input target 10 is placed on a rotational mount 11 • A pattern 
projecting section 12 projects a pattern light onto the target 10. 
The target with the pattern projected is captured by a light 
5 receiving section 13. 

The pattern pro ject ion and capturing are controlled by a 
controlling section 14. The captured image data is sent to a 
processing section 15. At the processing section, a three- 
dimensional shape calculation is performed, and the obtained 
10 three-dimensional shape data is stored in a recording section 16. 

Generally, an analog camera or digital camera is used as the 
light receiving section 13, but a special-purpose camera which 
obtains a non-visible light image can also be used depending on 
the target. Also, in general, a calculation unit such as a personal 
15 computer or a workstation is used as the controlling section 14 
and the processing section 15. Any recording medium can be used 
as the recording section 16 such as, for example, hard disk drive, 
CD-R, floppy disk, MO, and DVD, or the like. 

Fig. 2 roughly shows a flow of the process according to the 
20 present invention. The process includes a total of four steps, 
including the steps of calibration, rough shape extraction, 
detailed shape extraction, and three-dimensional shape data 
production . 

At the calibration step (S20), the internal parameters (for 
25 example, view angle information, that is, the ratio between the 
focal length and CCD size or lens distortion) and external 
parameters (camera positions) of the cameras are determined. At 
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the rough shape extraction step (S21), rough shape information of 
the target is obtained. At the detailed shape extraction step 
(S22) , on the other hand, detailed shape information of the target 
is obtained. At the three-dimensional shape data production step 
5 . (S23), the rough and detailed shape information are integrated to 
produce final three-dimensional shape data. 

An embodiment of the present invention will now be described 
in more detail. 

^ It is assumed that the target 10 is placed on the rotational 

ijj; 10 mount 11 and images of the target 10 are captured by a camera while 
the rotational mount 11 is rotated. It is also possible to place 
\r a plurality of cameras 13 around the target 10 and obtain a plurality 

of images of the target 10 from these cameras. The configuration 
* with a plurality of cameras is preferable when the target 10 is 

15 an object which cannot be placed on the rotational mount 11 or an 
object which cannot remain stationary for a long period of time. 

It is also assumed that the shape measurement is taken using 
the silhouette method at the rough shape extraction step and using 
an active stereo method by the space coding method using binary 
20 patterns at the detailed shape extraction step. Therefore, two 
cameras are used as the light receiving section 13. 

The embodiment will be described following the flow shown in 
Fig. 2. 

( Calibration step ) 
25 At the calibration step, the internal and external parameters 

of the cameras 13 are determined. The internal parameters of a 
camera 13 include the view angle information (a ratio between the 
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focal length and CCD size, which will be referred to as a 
"perspective ratio" hereinafter) and lens distortion parameter. 
The external parameters of the cameras 13 include the position and 
orientation information of the cameras 13 , and more specifically, 
the positional relationship between the rotational axis of the 
rotational mount and the cameras 13 , and the positional 
relationship between the cameras 13. 

Fig. 3 is a diagram for explaining the perspective ratio. A 
pinhole camera model is assumed as the camera model. The 
perspective ratio is defined as the ratio between the focal length 
and the CCD size (for example, the vertical length of CCD). In 
the case shown in Fig. 3, the perspective ratio would be h ccd /f . 
This corresponds to the parameter for determining 
enlargement/reduction ratio when the size of the object projected 
onto the screen is enlarged/reduced in perspective depending on 
the distance from the viewpoint to the object. 

In reality, it is more likely that the focal length of the 
camera 13 and the CCD size are unknown, or, even when these 
parameters are known, they may not be sufficiently accurate for 
use in actual shape measurement. 

In such a case, the perspective ratio can be determined as 
follows. First, a reference pole 30 of length H is captured so 
that the image fits in an image surface 31. The distance L between 
the camera and the reference pole 30 is then measured. If the 
vertical size of the capturing surface is h px (in units of pixels) 
and the reference pole is captured with a length of h pixels in 
the capturing surface, the perspective ratio can be calculated by 
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(H x h px )/(h x L) using the relation H/L = h/f. 

The method for determining the external parameter, that is, 
the positional relationship between the rotational axis of the 
rotational mount 11 and the camera 13 will now be described. This 
process will be referred to as "rotational mount calibration" 
hereinafter . 

Referring to Fig. 4, in the rotational mount calibration 
process, an object 40 (referred to as a "reference object") with 
a know pattern printed is placed on the rotational mount 11. The 
known pattern is, for example, circle markings 42, and a plurality 
of such known patterns are printed on a flat reference object. All 
of the circle markings are printed at pre-specif ied locations, and 
the printed positions of the circle markings are pre-registered 
at the calculation unit for performing the rotational mount 
calibration process. The positional relationship between an 
object with a known pattern printed and the camera 13 can be reliably 
determined using Hough transform. The specifics of Hough 
transform will be described later. 

In order to determine the positional relationship between the 
rotational axis 41 of the rotational mount 11 and the camera 13, 
it is not sufficient to only determine the positional relationship 
between the camera 13 and the reference object 40. To this end, 
the rotational mount 11 is rotated so that a plurality of images 
are taken for the reference object 40 from several points of view 
and the positional relationship between the camera 13 and the 
reference object 40 is determined for each viewpoint. Considering 
the fact that the camera viewpoint is fixed, this process 
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corresponds to obtaining information on how the reference object 

40 is moved with the rotational movement of the rotational mount 
11. This movement forms a circular motion around the rotational 
axis 41 of the rotational mount 11, and thus, the movement of an 

5 arbitrary point on the reference object 40 (for example, its 
origin) is tracked and a circle is fitted to the trajectory. A 
line through the center of this circle which perpendicularly 
intersect the plane formed by the circle is the rotational axis 

41 of the rotational mount 11. 

10 The positional relationship determination method between the 

reference object 40 and the camera 13 using the Hough transform 
will now be described. Referring now to Figs. 4 and 5, the 
reference object 40 is captured. The circle markings 42 are 
extracted from the captured image 50. A plurality of circle 

15 markings 42 are provided on the reference object 40 and thus, the 
circle markings cannot be distinguished from each other. Three 
circle markings are arbitrarily selected from among the extracted 
circle markings. Three circle markings data are also arbitrarily 
selected from the circle markings position data 51 registered at 

20 the calculation unit. 

A conversion matrix from the reference object coordinate 
system to the camera coordinate system is then determined assuming 
these data are correct. The conversion matrix can be considered 
as the camera position and orientation seen in the reference object 

25 coordinate system, and can be determined by six degrees of freedom, 
camera positions X, Y, and Z and camera directions, a , j3 , and y . 
The following equation 1 shows the relationship between the camera 
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coordinate system and the reference object coordinate system 
defined in terms of X, Y, Z, a, /? , and 7. 

(Equation 1) 

5 

Obviously, the correctness of the values X, Y, Z, a , j3 , and 
7 are not ensured. However, when the values X, Y, z, a, /3 , and 
7 are calculated for all combinations of arbitrarily selected 
q three circle markings and three registered positional data, some 

10 of the combination will be correct, and yield accurate values for 
77 X, Y, Z, a, (3 , and 7 . All of the correct combinations will yield 

J the same values for X, Y, Z, a , j3 , and 7 . The incorrect 

combinations, on the other hand, will yield random values for X, 
t Y, Z, a , j3 , and 7 . 

I 15 As shown in Fig . 6 , tables each corresponding to one of the 

variables X, Y, Z, a, /?, and 7 are prepared and all the values 
of X, Y, Z, a, /3 , and 7 obtained for all the combinations are 
polled in the table. The value that forms a peak in each table 
is considered as the valid value. A point in Fig. 6 pointed by 
20 an arrow in the upward direction represents the valid value for 
that variable. 

The positional relationship between the cameras 13 shown in 
Fig. 4, on the other hand, can be easily calculated from the 
conversion matrices between the reference object coordinate system 
25 and the camera coordinate systems when the same reference objet 
40 is captured by a plurality of cameras 13. Specifically, if the 
value of the three-dimensional point P, of camera a coordinate 
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system, represented in the coordinate system of camera a is Pa, 
the value of the same three-dimensional point P represented in the 
coordinate system of camera b is Pb, and the value of P in the 
reference object coordinate system is Po, the relationship between 
5 these values will become: 
P a = R oa * P 0 + T oa 
P b = R ob * P c + T ob 

Where R^, R ob , T oa , and T ob are respectively rotation and translation 
parameters from the reference object coordinate system into the 
10 coordinate system of camera a or camera b. The coordinate 
conversion between the cameras will be: 
P a = Roa * Rob (_1> * P b + (T oa - Rob (_1) * T ob ) 

Thus, the origin of camera b, for example, is at a position (T oa 
- R 0 b ( ~ 1) T ob ) in the camera a coordinate system. 

15 (Rough shape calculation step by silhouette method) 

The rough shape extraction using the silhouette method will 
now be described. Referring to Fig. 7, a process for extracting 
silhouette of the target is performed on a plurality of object 
images input by the cameras (S70) . For example, a chroma key method 

20 using blue back or background subtraction method where the 
background image data is pre-input and silhouette is obtained by 
difference processing the object image data and background image 
data can be used for silhouette extraction. The detailed 
description of the chroma key method and the background subtraction 

25 method will not be given here, but appears in, for example, Japanese 
Patent Laid-Open Publication No. Hei 11-96361. 

At the next step, a shape reconstruction process is performed 
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to reconstruct the three-dimensional shape of the target from the 
plurality of silhouette images (S71) . In this step, the 
three-dimensional space where the target object is expected to 
exist is represented as voxel space. The voxel space is a space 
5 that represents the three-dimensional space by a group of tiny 
three-dimensional elements (voxels) . The space where the target 
object exist can be expected to be is the region above the rotational 
mount and which can be fitted into the camera view. 
^ Each voxel is then projected onto an individual silhouette 

g! 10 image and only the voxels that are projected onto all the object 
J7 silhouette images (or onto at least a predetermined number of 

silhouette images) are extracted as the rough shape of the object. 
l~ Fig. 8 explains the shape reconstruction process as described 

=1? above. In general, projection process to the voxel space 8 0 is 

y 15 performed based on multiple silhouette images, but the figure only 
shows a portion of the silhouette images. 

At this point, a possible existence region is considered. The 
possible existence region for silhouette image 1 is the conical 
region 82 having the center of the camera viewpoint 81 as its vertex 
20 and the object image within the silhouette image 1 as its cross 
sectional shape. The possible existence regions for other 
silhouette images can be similarly defined. Thus, a target object 
will necessarily exist (unless there is an error in the silhouette 
images) inside the possible existence region. This process can 
25 be considered to correspond to an operation to cut the portions 
outside the silhouette from a gypsum block and finally cut out the 
object. 
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The group of voxels extracted in this manner represents the 
three-dimensional shape of the object. However, detail of the 
ob j ect shape cannot be represented by the rough shape . For example , 
care must be taken in a portion which cannot be observed as a 
5 silhouette outline (for example, the inside of a cup) , because the 
rough shape will be calculated in a condition with the portion 
filled in. One method for accurately calculating such a convex 
shape is a space coding method which will be described next. 
(Detailed shape calculation step using space coding method) 
fn 10 An active stereo method by space coding method using binary 

Cj patterns will now be described. A binary pattern refers to a 

^ black-and-white pattern. One set of examples of coded binary 

patterns are shown in Figs. 9(A) through 9(G). Fig. 9(A) shows 
i a pattern having a black half and a white half. The pitch is 

I 15 sequentially halved for the patterns shown in Fig. 9(B) , 9(C) , ... 

9(G). Such pattern lights are sequentially projected onto the 
target . 

Fig. 10 shows, for descriptive purpose, an example of a code 
when three types of pattern lights are sequentially projected. If 
20 each pattern is considered as 8-bit data, the light portion of 
pattern A (shown as (A) in Fig. 10) is (1111) and the dark portion 
of pattern A is (0000) , and the overall pattern can be represented 
by 8 bits, (11110000). Similarly, pattern B (shown as (B) in Fig. 
10) is (11001100) and pattern C (shown as (C) in Fig. 10) is 
25 (10101010). 

Now, if pattern A is considered as an MSB (Most Significant 
Bit) and pattern C is considered as an LSB (Least Significant Bit) , 
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a 3-bit binary code can be assigned to each of the divided regions. 
For example, a number (000) = 0 ( in decimal) can be assigned to 
the right most region and a number (111) = 7 (decimal) can be 
assigned to the left most region. 
5 By using n patterns as described, an n-bit binary code can 

be assigned to a space region, and thus, the space can be identified 
at once by this code. In Fig. 9, because a total of seven patterns 
are used, 128 space regions can be identified at once. When the 
q space region of the point P where the object is located is specified 

10 by projecting such pattern lights onto the target, the space 
^Jj regions respectively projected onto the two camera images can be 

jTT easily identified, and the correspondence relationship between the 

I images can be easily obtained. 

In other words, sections, for example, that are observed as 
U 15 0010101 in time sequence in an image is unique for both camera images 
"rf and they constitute correspondence points . Because the two camera 

positions are already determined at the calibration step, the 
three-dimensional position of each space region can be easily 
calculated by the principle of triangulation . 
20 The detailed shape obtained in this manner becomes the depth 

data from the observation viewpoint, that is, the viewpoint of the 
light receiving section, and referred to as "detailed depth map" 
hereinafter. The detailed depth map is used for the three- 
dimensional shape data production at the next step. Details of 
25 the three-dimensional position measurement by space coding is 
described in, for example, "Three-dimensional image measurement" 
(by Masashi Iguchi and Hirosuke Sato, Shokodo) . 
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(Three-dimensional data production step) 

The detailed shape is determined by the space coding method 
as described above, but, for a black portion where light is absorbed, 
there are some cases where no information can be obtained. In 
5 contrast, although detailed shape is not determined by the 
silhouette method, rough shape can be determined for the black 
portion where light is absorbed. Therefore, it is necessary to 
integrate the produced data to combine the advantages of the two 

pi methods. The integration process is performed at the three- 

^ 10 dimensional shape data producing step. 

Zs At the three-dimensional shape data producing step, first, 

*f the rough shape represented by the voxels is converted into depth 

data from the camera viewpoint in the space coding method. This 
J depth data is referred to as a "rough depth data" hereinafter, 

f 15 There also exist detailed depth map data obtained by the space 

coding method. 

There is a possibility that a portion of the information is 
absent in the detailed depth map when a black portion which absorbs 
light is present in the target. In the rough depth map, on the 
20 other hand, there is a possibility that detailed concavity is not 
represented. Thus, the condition between each of the depth map 
information for each section can be divided into the following four 
cases: 

1. the depth value of the rough and detailed depth maps are equal; 
25 2 . the depth value of the detailed depth map is larger than that 
of the rough depth map; 
3. the depth value of the detailed depth map is smaller than that 
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of the rough depth map; or 
4. no value exists in the detailed depth map. 

The first case corresponds to a section where the object 
shape is accurately measured by both silhouette and space coding 
5 methods. The second case corresponds to a section where the object 
shape is more accurately measured by the space coding method than 
by the silhouette method. The third case, on the other hand, 
corresponds to a case where an obviously erroneous value is output 
f! by the space coding method due to failure of black-and-white 

lg 10 pattern extraction (specifically, determination that white 
JJ pattern is projected even though a black pattern is projected or 

^ the like) in the space coding method. 

This is because a true object surface always exists on or 
inside a surface determined by the silhouette method. The fourth 
f7 15 case corresponds to a section where no shape data is obtained due 
^ to reasons such as the section being black. 

Considering the above, a final depth map is obtained by 
integrating the two depth maps as follows. 

For cases 1 and 2: Use the detailed depth map as the final depth 
20 map. 

For cases 3 and 4: Use the rough depth map as the final depth map. 

Fig. 11 shows a bird's eye view (wire frame display) of the 
rough and detailed depth maps and the final depth map obtained from 
these depth maps. 
25 The final depth map is then applied on the rough data which 

is already obtained (and represented in the form of a voxel group) . 
In other words, among the voxels that represent the rough data, 
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the voxels that are not consistent with the final depth map, more 
specifically, the voxels with shorter distance than the distance 
from the light receiving section viewpoint indicated in the final 
depth map, are removed. After such removal, the object shape is 
5 represented by the remaining voxels. 

The three-dimensional shape data is often used in various 
tools such as CG (Computer Graphics) and CAD (Computer Aided 
Design). During such processes, it is more typical to represent 
q the three-dimensional shape data in a parametric surface such as 

^ 10 polygon or NURBS, rather than by voxels. For this purpose, it is 
^ possible to extract voxels that correspond to the object surface 

£v from the group of voxels, and to convert the data into a polygon 

r~ representation or into NURBS by connecting the surface voxels. 

1% The following is a summary that repeats the effect of the 

fr 15 method as described above. 

£7 !• In the silhouette method, three-dimensional shape information 

can be obtained irrespective of the color of the target, but 
convex shape portions cannot be accurately measured. 

2. A triangulation method using a laser light and a space coding 
20 method for projecting a plurality of black-and-white patterns 

are available for accurately determining the shape of the 
convex portions, but in these methods, data cannot be obtained 
for black color portions and for mirror reflective sections. 

3. By combining the rough shape extraction by the silhouette 
25 method and, for example, space coding, it is possible to obtain 

good three-dimensional shape measurements for both convex 
shapes and black and mirror reflective sections. 
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(Another embodiment: reduction in the number of patterns) 

The above description described a method to realize a 
general-purpose three-dimensional shape modeling by suitable 
usage of the advantages for the silhouette and space coding methods . 
It is also possible to reduce the number of the projection patterns 
at the space coding method by employing the rough shape information 
by the silhouette method or the like. An embodiment using this 
advantage will now be described. 

A conceptual structure in this embodiment is identical to that 
of the first embodiment and will not be described again. A rough 
flow is also identical to that of the first embodiment and includes 
four steps , a calibration step, a rough shape extraction step, a 
detailed shape calculation step, and a three-dimensional data 
producing step. In the detailed shape calculation step, however, 
target surface candidate extraction is performed. The three- 
dimensional data producing step includes two substeps, a step for 
determining the possible existence region for a surface and a step 
for determining the final three-dimensional shape. This is shown 
in Fig. 12 . The calibration and rough shape extraction are already 
described, and will not be described again. 

The process at the target surface candidate extraction step 
will now be described. Consider a case where only patterns shown 
in Figs. 9(D), 9(E), 9(F), and 9(G) are used from among the patterns 
shown in Fig. 9. In this case, 3-bits worth of information is 
absent which would have been distinguished by patterns (A), (B), 
and (C), and in general, eight possible points are present which 
show the same black-and-white pattern sequence. 
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Fig. 13 shows the condition described above. Here, two 
cameras a and b are placed at the right and left, and vertical binary 
patterns are projected onto the target. The figure is a view seen 
from the top. Assume a code 0010 is assigned at a point on the 
5 screen of camera a. The points on the screen of camera b which 
have the same code 0010 are shown by circles, which are referred 
to as candidates for corresponding point. Eight such 
correspondence candidate points exist. The three-dimensional 
m positions of these eight points are indicated by circles in the 

10 figure shown as a surface candidate. These eight points are 
therefore extracted as the object surface candidate. 
;? The step for determining a possible existence region for a 

*~ surface will now be described . The inside region of the rough shape 

Z can be considered to indicate a region where the surface of the 

f 15 object can exist. This is because no surface of the object can 
exist outside the rough shape. In addition, when some 
characteristics of the target are known, a further limitation can 
be applied to the possible existence region for a surface. For 
example, when a human face is input, there are certain limitations 
20 on the asperity of the surface. 

Thus, it is possible to apply a rule, such as, for example, 
"no true surface exists 3 cm inside of the rough shape surface" . 
Such a rule is not only applicable to a human face but also to a 
general shape when some characteristics of the target object are 
25 known. Therefore, at the step for determining a possible existence 
region for a surface, a region where the surface of the object can 
exist is determined based on the rough shape information and some 
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characteristics of the target object. 

At the final three-dimensional shape extraction step, only 
the obtained object surface candidates that exist within the 
possible existence region for the surface are extracted, in the 
5 case shown in Fig. 13 , for example, although eight object surface 
candidates are originally present, the distance between the 
candidates are large. If the determined possible existence region 
for a surface is the portion indicated by the slashes, the only 
candidate among the eight candidates that fits in the possible 
^10 existence region for a surface is the candidate indicated by the 
^ reference numeral 130. Thus, the circle shown by the reference 
4* numeral 130 is extracted as the final three-dimensional shape. 
O In this manner, by using the rough shape information, the 

number of patterns at the space coding method can be reduced. This 
1315 is especially useful in reducing the length of the time for 
O inputting data. When a triangulation method using laser light 
O irradiation is used in place of the space coding method, noise 
related to the target object surface data (section of the surface 
shape data that is erroneously measured as extremely protruding 
20 from or lowered compared to the surface) which is often referred 
to as a spike can be easily removed. 
(Date recording) 

The data production has been described. When new three- 
dimensional data is produced by integrating three-dimensional data 
25 obtained by different devices or methods having different 
reliability or precision, the data will have different precision 
or reliability depending on the section. 
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Foe example , the section where the rough shape data is taken 
as the final data can be considered as having lower reliability 
than the section where the detailed shape data is used as the final 
data. It is therefore preferable to record the data with the 
5 reliability evaluation value for each section of the three- 
dimensional data. 

An example of reliability data is shown in Fig. 14 where the 
three-dimensional data is to be represented based on polygons. As 
shown in Fig. 14, the three-dimensional data can be divided into 
f^lO a vertex coordinate data section and the polygon data section where 
gj. the polygons are constructed by connecting the vertices. For 
T\ example, the data indicates that the Oth vertex (vid 0) is located 
q at (x, y, z) = (0.0, 20.0, 43.0), the first vertex is located at 
^ (10.3, 5.2, 133.8), and the second vertex is located at (-20.9, 

rf!5 3.2, 13.9), and the Oth polygon (pidO) is constructed by connecting 
^ the Oth, first, and second vertices . The reliability (for example, 
jj«f defined to have higher values for higher reliability) is defined 
for each vertex, for example r = 1.0 is defined for the Oth vertex 
and r = 0.5 for the first vertex. 
20 In this manner, by recording the reliability data along with 

the three-dimensional shape data, it is possible to proceed with 
the integration process while weighting the data based on the 
reliability to integrate the produced three-dimensional shape data 
and three-dimensional shape data of the same target but obtained 
25 by different method or device. 

When the three-dimensional shape data is applied in various 
applications, the portion that can be used can be selectively 
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extracted based on the reliability. For example, when the shape 
is to be viewed on the Internet, the precision of the shape is not 
as important, and the three-dimensional shape data can be shown 
including the data with low reliability. 
5 When the same data is input to a CAD tool, on the other hand, 

the precision of the shape is important, and only the data with 
high reliability is input. The other data can be interpolated and 
produced by a manual modeling function provided in the CAD tool. 
Moreover, when the three-dimensional data is to be manually 

10 corrected and processed, it is generally true that the 
maneuverability improves if the points to be corrected are 
automatically told to an operator. In this case, if the 
reliability information is added to the three-dimensional data, 
the correction points can be told based on this information. 

15 The above description of the data recording assumes that the 

reliability is evaluated in a numerical value. However, the 
evaluation of the reliability is not limited to a numerical value. 
There can be another method, for example, where the reliability 
is divided into several levels, and a symbol is assigned to each 

20 of the levels. In case where the data recording method is a data 
recording method integrating a plurality of media, a method can 
be used where the reliability is represented visually or by audio. 
(Others) 

In the above description, the rough shape is determined by 
25 the silhouette method, but the rough shape can be obtained by other 
methods. When the choice of a target is narrowed to some extent 
in advance, it is also preferable to give the rough shape by manually 
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supply or by supply from database or the like. 

Also, in the above description , an active stereo method using 
a binary coded light is used as the method for the detailed shape 
extraction. The detailed shape extraction can also be performed 
by a triangulation method using color coding light or a laser light, 
or by a Moire method. The "pattern light'' in the present invention 
includes the projection lights used in these methods. If the 
positional relationship between the pattern projecting section and 
the camera is accurately known in advance, the number of cameras 
to be used can be one. 

It is also described that it is possible to reduce the number 
of projection patterns in the space coding method using the rough 
shape information. Similarly, in the triangulation method using 
a laser light, it is possible to irradiate a plurality of laser 
lights, obtain a plurality of candidates for depth data, and then 
narrow the candidates using the rough shape information. In such 
a case, the data input can be accelerated compared to a case where 
only a single laser light is used. 

In addition, in the above, the rough shape obtained by the 
silhouette method is represented as a group of voxels. However, 
it is also possible to reconstruct the shape by an envelope line, 
as described in "Shape Measurement and Description of Curved 
Surface Objects", Sato et al., Electrical Communication Institute 
Paper, Vol. J-62-D, No. 1( 1979). It is also possible to represent 
the reconstructed shape by a parametric surface such as polygons 
or NURBS. 

(Binary pattern projection which does not require position 
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coincidence ) 

A space coding method using general binary patterns is 
described above. 

When binary patterns as shown in Figs. 9 or 10 are used, the 
5 following problem is observed. In general, there is a need to 
separate the input image with the binary pattern projected onto 
the target into a portion corresponding to the white pattern (white 
pattern region) and a portion corresponding to the black pattern 
(black pattern region) . The boundary between the white pattern 

10 and black pattern is unstable in that it may be assigned to either 
the white pattern region or the black pattern region. 

Fig. 16 shows a condition where binary patterns are projected 
from the pattern projecting section 12 onto the input target 10 
and images are input by two cameras 13A and B. Fig. 17 shows an 

15 enlarged view near the boundary between the white and black pattern 
sections in the image input by camera A. 

Fig. 17 (al) shows a portion of the image input by camera A. 
Fig. 17(bl) shows a portion of the image input by camera B. In 
both cases, the boundary between the black and white patterns 

20 indicated by an arrow pointing upward has an intermediate lightness . 
In such a case, it is indecisive as to which of the two codes are 
finally assigned to the portion indicated by the up-pointing arrow, 
a code corresponding to the black pattern or a code corresponding 
to the white pattern . 

25 Fig. 17 (a2) shows a case where the portion in an image input 

by camera A indicated by the up-pointing arrow is assigned a code 
corresponding to the white pattern and Fig. 17 (b2) shows a case 
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where the portion in an image input by camera B indicated by the 
up-pointing arrow is assigned a code corresponding to the black 
pattern. In this case, although the same code should be assigned 
to the corresponding portions indicated by the up-pointing arrows , 
5 in reality, different codes are assigned, resulting in an error 
in the correspondence relationship. 

In order to avoid such a problem, there is a method, as shown 
in Fig. 18, in which the boundary i between the white and black 
in pattern A is provided at a point corresponding to the midpoint 
2 10 between the boundaries ii and iii between the white and black in 
1g pattern B. A similar positional relationship applies between the 

p patterns B and C. For the most detailed pattern, the code 

jy^ assignment is assigned to the pattern boundary. That is, the 

^ boundary having white at the left and black at the right is assigned 

~: 15 a code of 1 and the boundary with black at the left and white at 
!Z the right is assigned a code of 0. 

f ~' However, in this case, the pattern position must be accurately 

set. If the setting is not accurate, it is possible that, for 
example, pattern A will slide to the right, resulting in the 
20 boundaries i and ii coinciding, and thus, the same problem is 
generated as described above. 

However, in order to accurately set the pattern positions, 
an expensive pattern projecting mechanism will be required. When 
there is no need to accurately set the pattern positions, on the 
25 other hand, an inexpensive pattern projecting mechanism such as 
a slide projector can be used. In order to avoid the above 
described problem without accurately setting the pattern positions , 
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it is preferable to employ the following method. 

The method classifies the input image with the binary patterns 
projected onto the target into six types of regions as shown in 
Fig. 19 , rather than into only two, the portion corresponding to 
the white pattern (white pattern region) and the portion 
corresponding to the black pattern (black pattern region) . 
Specifically, the six types include: 
A: white pattern region; 

B: white region at the transition region from the white pattern 
to the black pattern; 

C: black region at the transition region from the white pattern 

to the black pattern; 

D: black pattern region; 

E: black region at the transition region from the black pattern 
to the white pattern; and 

F: white region at the transition region from the black pattern 
to the white pattern. 

A correspondence relationship is then obtained after such 
separation into regions (code assignment) . 

Here, the following correspondence relationships are taken 
to be true in order to consider errors in region assignment. 
A <-> F, A, B 
B <-> A, B, C 
C <-> B, C, D 
D <-> C, D, E 
E <-> D, E, F 
F <-> E, F, A 
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In this manner, by increasing the number of assigned codes 
from two and introducing an allowance range for the correspondence 
relationship, the correspondence relationship can be stably 
obtained even when there are some unstable elements during the 
5 region separation. 

(Color code pattern projection) 

A space coding method using a color pattern will be described 
now. In the above space coding method using binary patterns, a 
plurality of patterns need to be projected. Alternatively, by 
q 10 employing a color pattern, the number of patterns to be projected 
g can be reduced. 

Jj Fig. 20 shows an example of a color pattern. It is preferable 

2 to have the colors between the adjacent patterns be as different 

' as possible. It is also preferable that the colors, when 

"ri 15 represented in HSV space, have at least 90 degrees difference in 
M hue, or at least 0.3 difference in lightness. In the case shown 

in Fig. 20, the pattern is constructed using four colors, red, green, 
blue, and black. It is also possible to use other combinations 
or other colors with sufficient differences in hue and lightness. 
20 The distance from the viewpoint where each image is captured 

to the target is calculated for an image of an object projected 
with such a pattern by the following processes . 

1 . Extract characteristic points from the image 

Edge extraction is performed on the image and a thinning 
25 operation is applied. Thinned edges having a value of at least 
a predetermined value are considered as characteristic points. 

2. Assign a property to each characteristic point 
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The color information at the left and at the right of the 
characteristic point is assigned as a property. In order to 
stabilize the property assignment, it is preferable to average the 
color data within a region between right and left characteristic 
5 points. Also, it is possible to average the color data within a 
region after region division is applied to the image. 
3. Associate characteristic points between images from a 
plurality of viewpoints based on the property 

Association between characteristic points extracted from 
10 individual images is performed based on the property. 

Here, it is assumed that the correspondence is more probable 
for characteristic points having similar property. The 
similarity is calculated by representing the property by RGB and 
using an equation: 
15 |Rla - Rib | + |Gla - Gib | + |Bla - Blb| + |Rra - Rrb | + | Gra - Grb | 
+ | Bra - Brb| 

Here, Rla means the value of the R component related to the color 
at the left of the characteristic point in the image from camera 
viewpoint a, and | Rla - Rib | means the absolute value of the 
20 difference between Rla and Rib . Alternatively, it is also possible 
to similarly evaluate the similarity using HSV or other color 
spaces . 

In process 3, it is also preferable to use a multocular stereo 
method of trinocular or more in place of a binocular stereo method 
25 where characteristic points in two images having the most similar 
properties are associated. The multocular stereo method is more 
reliable in associating the characteristic points. 
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For example , in a trinocular stereo method, in addition to 
extracting a plurality of characteristic points that are similar 
in the property between two images (AB pair) corresponding to 
camera viewpoints A and B, a plurality of characteristic points 
are also extracted for two images corresponding to camera 
viewpoints B and C (BC pair) which have similar property. For each 
corresponding characteristic point candidate , a depth value from 
the camera viewpoint B is determined. If the corresponding 
characteristic point candidate is, in fact f the corresponding 
characteristic point, the depth value determined from the AB pair 
and the depth value determined from the BC pair should coincide 
(or be within an error limit) . 

Therefore, the depths for extracted candidate characteristic 
points are compared, and the candidate with the smallest difference 
is assumed to be the true corresponding point. In this manner, 
the correspondence of the characteristic points can be reliably 
determined. Fig. 21 shows the depth value for correspondence 
characteristic point candidates using three cameras, A, B, and C. 
In this case, the pair indicated by a circle will be ultimately 
judged to be the true corresponding characteristic point. 

In the above description, the stereo camera positions are 
assumed in the horizontal direction (right and left) . Accordingly, 
the patterns are assumed to be vertical stripes. When the stereo 
cameras are positioned vertically (above and below), it is 
preferable to use horizontal stripe patterns. 
(Usage of 3D sensors with different precision) 

In addition, it is also preferable to use a plurality of 
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three-dimensional shape inputting means, for example, laser 
scanner heads, each with different input precision in the 
three-dimensional shape inputting apparatus . 

In general, a laser scanner head with high input precision 
5 has smaller measurement range than a laser scanner head with lower 
input precision. Using this general fact, the rough shape of the 
target is first obtained using the lower precision laser scanner 
head. More detailed target shape can then be obtained using the 
higher precision laser scanner head. 
Q 10 Here, the high precision laser scanner head is mounted on a 

CO robot arm or on a multi-axis stage which moves the high precision 

%| laser scanner head to a suitable position for measurement based 

M on the obtained rough shape in order to obtain necessary shape data . 

h: In this manner, it is possible to measure the shape automatically 

In 15 and with high precision for a target having a large size such that 
|^ it is difficult to measure the entire target by the high precision 

laser scanner head alone. 

In the above description, a case is shown where two types of 
laser scanner heads with different precision are used, but it is 
20 also possible to use an increased number of laser scanner heads. 
Also, it is obviously possible to use other three dimensional shape 
measurement means based on other methods in place of the laser 
scanner heads, such as, for example, three-dimensional measurement 
means using an active stereo method which is already described or 
25 a Moire method. 

As described, according to the present invention, by using 
both rough shape information and detailed shape information 
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obtained by light irradiation, it is possible to obtain good 
three-dimensional shape measurements for both convex shape and 
dark colored section and mirror reflective section. 

Moreover, by using the rough shape information, it is possible 
5 to reduce the number of projection patterns in the space coding 
method or to easily remove the spike noise in the triangulation 
method using a laser light irradiation. 



T 7 * 
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